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RELATED APPLICATIONS 
[0001] This application is related to, and claims priority from, U.S. Provisional 

Patent Application Serial No. 60/566,444 filed on April 30, 2004, entitled "Freespace 
Pointing Device", the disclosure of which is incorporated here by reference. This application 
is also related to, and claims priority from, U.S. Provisional Patent Application Serial No. 
60/612,571, filed on September 23, 2004, entitled "Free Space Pointing Devices and 
Methods", the disclosure of which is incorporated here by reference. This application is also 
related to, and claims priority from, U.S. Provisional Patent Application Serial No. 
60/641,383, filed on January 5, 2005, entitled "Methods and Devices for Removing 
Unintentional Movement in Free Space Pointing Devices", the disclosure of which is 
incorporated here by reference. This application is also related to U.S. Patent Applications 

Nos. , , and , entitled "Free Space 

Pointing Devices With Tilt Compensation and Improved Usability", "Methods and Devices 
for Identifying Users Based on Tremor", and "Free Space Pointing Devices and Methods", all 
of which were filed concurrently here with and all of which are incorporated here by 
reference. 
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BACKGROUND 

[0002] The present invention describes techniques and devices for processing motion 

data associated with a device, e.g., a handheld device, to remove unintentional movement 
associated therewith. According to some exemplary embodiments of the present invention, 
the handheld device can be a free space pointing device. 

[0003] Technologies associated with the communication of information have evolved 

rapidly over the last several decades. Television, cellular telephony, the Internet and optical 
communication techniques (to name just a few things) combine to inundate consumers with 
available information and entertainment options. Taking television as an example, the last 
three decades have seen the introduction of cable television service, satellite television 
service, pay-per-view movies and video-on-demand. Whereas television viewers of the 
1960s could typically receive perhaps four or five over-the-air TV channels on their 
television sets, today's TV watchers have the opportunity to select from hundreds, thousands, 
and potentially millions of channels of shows and information. Video-on-demand 
technology, currently used primarily in hotels and the like, provides the potential for in-home 
entertainment selection from among thousands of movie titles. 

[0004] The technological ability to provide so much information and content to end 

users provides both opportunities and challenges to system designers and service providers. 
One challenge is that while end users typically prefer having more choices rather than fewer, 
this preference is counterweighted by their desire that the selection process be both fast and 
simple. Unfortunately, the development of the systems and interfaces by which end users 
access media items has resulted in selection processes which are neither fast nor simple. 

-3- 
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Consider again the example of television programs. When television was in its infancy, 
determining which program to watch was a relatively simple process prunarily due to the 
small number of choices. One would consuh a printed guide which was formatted, for 
example, as series of columns and rows which showed the correspondence between (1) 
nearby television channels, (2) programs being transmitted on those channels and (3) date and 
tune. The television was tuned to the desired channel by adjustmg a tuner knob and the 
viewer watched the selected program. Later, remote control devices were introduced that 
permitted viewers to tune the television from a distance. This addition to the user-television 
interface created the phenomenon known as "channel surfmg" whereby a viewer could 
rapidly view short segments being broadcast on a number of channels to quickly learn what 
programs were available at any given time. 

[0005] Despite the fact that the number of channels and amount of viewable content 

has dramatically increased, the generally available user interface, control device options and 
frameworks for televisions has not changed much over the last 30 years. Printed guides are 
still the most prevalent mechanism for conveying programming information. The multiple 
button remote control with up and down arrows is still the most prevalent channel/content 
selection mechanism. The reaction of those who design and implement the TV user interface 
to the increase in available media content has been a straightforward extension of the existing 
selection procedures and mterface objects. Thus, the number of rows in the printed guides 
has been increased to accommodate more channels. The number of buttons on the remote 
control devices has been increased to support additional functionality and content handling, 
e.g., as shown in Figure 1. However, this approach has significantly increased both the time 
required for a viewer to review the available information and the complexity of actions 
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required to implement a selection. Arguably, the cumbersome nature of the existing interface 
has hampered commercial implementation of some services, e.g., video-on-demand, suice 
consumers are resistant to new services that will add complexity to an interface that they 
view as already too slow and complex. 

[0006] In addition to increases in bandwidth and content, the user interface bottleneck 

problem is being exacerbated by the aggregation of technologies. Consumers are reacting 
positively to having the option of buying integrated systems rather than a number of 
segregable components. An example of this trend is the combination televisionA^CR/DVD 
in which three previously independent components are frequently sold today as an integrated 
unit. This trend is likely to continue, potentially with an end result that most if not all of the 
communication devices currently found in the household will be packaged together as an 
integrated unit, e.g., a televisionA^CR/DVD/intemet access/radio/stereo unit. Even those 
who continue to buy separate components will likely desire seamless control of, and 
interworking between, the separate components. With this increased aggregation comes the 
potential for more complexity in the user interface. For example, when so-called "universal" 
remote units were introduced, e.g., to combine the functionality of TV remote units and VCR 
remote units, the number of buttons on these universal remote units was typically more than 
the number of buttons on either the TV remote unit or VCR remote unit individually. This 
added number of buttons and functionality makes it very difficult to control anything but the 
simplest aspects of a TV or VCR without hunting for exactly the right button on the remote. 
Many times, these universal remotes do not provide enough buttons to access many levels of 
control or features unique to certain TVs. In these cases, the original device remote unit is 
still needed, and the original hassle of handling multiple remotes remains due to user 
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interface issues arising from the complexity of aggregation. Some remote units have 
addressed this problem by adding "soft" buttons that can be programmed with the expert 
commands. These soft buttons sometimes have accompanying LCD displays to indicate their 
action. These too have the flaw that they are difficult to use without loolcing away from the 
TV to the remote control. Yet another flaw in these remote units is the use of modes in an 
attempt to reduce the number of buttons. In these "moded" universal remote units, a special 
button exists to select whether the remote should communicate with the TV, DVD player, 
cable set-top box, VCR, etc. This causes many usability issues including sending commands 
to the wrong device, forcing the user to look at the remote to make sure that it is in the right 
mode, and it does not provide any simplification to the integration of multiple devices. The 
most advanced of these universal remote units provide some integration by allowing the user 
to program sequences of commands to multiple devices into the remote. This is such a 
difficult task that many users hire professional installers to program their universal remote 
units. 

[0007] Some attempts have also been made to modernize the screen interface between 

end users and media systems. However, these attempts typically suffer from, among other 
drawbacks, an inability to easily scale between large collections of media items and small 
collections of media items. For example, interfaces which rely on lists of items may work 
well for small collections of media items, but are tedious to browse for large collections of 
media items. Interfaces which rely on hierarchical navigation (e.g., tree structures) may be 
speedier to traverse than list interfaces for large collections of media items, but are not readily 
adaptable to small collections of media items. Additionally, users tend to lose interest in 
selection processes wherein the user has to move through three or more layers in a tree 
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structure. For all of these cases, current remote units make this selection processor even 
more tedious by forcing the user to repeatedly depress the up and down buttons to navigate 
the list or hierarchies. When selection skipping controls are available such as page up and 
page down, the user usually has to look at the remote to find these special buttons or be 
tramed to know that they even exist. Accordingly, organizing frameworks, techniques and 
systems which simplify the control and screen interface between users and media systems as 
well as accelerate the selection process, while at the same time permitting service providers 
to take advantage of the increases in available bandwidth to end user equipment by 
facilitating the supply of a large number of media items and new services to the user have 
been proposed in U.S. Patent Application Serial No. 10/768,432, filed on Januairy 30, 2004, 
entitled "A Control Framework with a Zoomable Graphical User Interface for Organizing, 
Selecting and Launching Media Items", the disclosure of which is incorporated here by 
reference. 

[0008] Of particular interest for this specification are the remote devices usable to 

mteract with such fi-ameworks, as well as other applications and systems. As mentioned in 
the above-incorporated application, various different types of remote devices can be used 
with such firameworks including, for example, trackballs, "mouse"-type pointing devices, 
light pens, etc. However, another category of remote devices which can be used with such 
frameworks (and other applications) is free space pointing devices. The phrase "free space 
pointing" is used in this specification to refer to the ability of an input device to move in three 
(or more) dimensions in the air in front of, e.g., a display screen, and the corresponding 
ability of the user interface to translate those motions directly into user interface commands, 
e.g., movement of a cursor on the display screen. The transfer of data between the free space 
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pointing device may be performed wirelessly or via a wire connecting tlie free space pointing 
device to another device. Thus "free space pointing" differs from, e.g., conventional 
computer mouse pointing techniques which use a surface, e.g., a desk surface or mousepad, 
as a proxy surface from which relative movement of the mouse is translated into cursor 
movement on the computer display screen. An example of a free space pointing device can 
be found in U.S. Patent No. 5,440,326. 

[0009] The '326 patent describes, among other things, a vertical gyroscope adapted 

for use as a pointing device for controlling the position of a cursor on the display of a 
computer. A motor at the core of the gyroscope is suspended by two pairs of orthogonal 
ghnbals from a hand-held controller device and nominally oriented with its spin axis vertical 
by a pendulous device. Electro-optical shaft angle encoders sense the orientation of a hand- 
held controller device as it is manipulated by a user and the resulting electrical output is 
converted into a format usable by a computer to control the movement of a cursor on the 
screen of the computer display. 

[0010] However, the freedom of use associated with free-space pointers creates 

additional challenges. For example, when a button is pressed on a free-space pointing device, 
the button actuation causes undesu-able movement of the device that should not be translated 
into pointer movement. One technique for addressing a portion of this problem' is to use a 
distance threshold to reactivate pointer movement after a button press occurs, i.e., to ignore 
any movement of the pointer after a button press occurs until the free space pointing device 
moves a predetermined distance. While simple, this technique may ignore intended 
movements of the device as well as unintended movements. 
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[0011] Accordingly, the present invention describes, among otlier things, methods 

and devices for processing pointer data on button clicks and other actions to remove 
undesirable movement without eliminating intended movement. 



SUMMARY 

[0012] According to one exemplary embodiment of the present invention, a method 

for operating a free space pointing device includes the steps of sensing movement of the free 
space pointmg device and generating an output associated therewith, detecting occurrence of 
an event, determining a velocity of the free space pointmg device, if the velocity of the free 
space pointing device is below a predetermined velocity threshold, then discarding or 
attenuating movement information sensed after the detected occurrence of the event and 
otherwise, if the velocity of the free space pointing device is above the predetermined 
velocity threshold, then filtering movement information sensed after the detected occurrence 
of the event to remove unintended motion associated with the user-initiated event. 
[0013] According to another exemplary embodiment of the present invention, a 
method for operating a free space pointing device includes the steps of sensing movement of 
the free space pointing device and generating an output associated therewith, detecting 
occurrence of an event and discarding or attenuating the output until a predetermined time 
has elapsed after the detection of the event. 

[0014] According to yet another exemplary embodiment of the present invention, a 

method for operatmg a free space pointing device includes the steps of sensing movement of 
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the free space pointing device and generating an output associated therewith, detecting 
occurrence of an event and filtering the output based on the detected event. 
[0015] According to a still forther exemplary embodiment of the present invention, a 

handheld pointing device includes at least one motion sensor for sensing movement of the 
handheld pointing device and generating an output associated therewith and a processing unit 
for detecting occurrence of an event and discarding or attenuating the output until either a 
predetermined time has elapsed after the detection of the event or the handheld pointing 
device has moved a predetermined distance after the detection of the event, whichever comes 
first. 

BRIEF DESCRIPTION OF TOE DRAWl^fGS 
[0016] The accompanying drawings illustrate exemplary embodiments of the present 

invention, wherein: 

[0017] FIG. 1 depicts a conventional remote control unit for an entertainment system; 

[0018] FIG. 2 depicts an exemplary media system in which exemplary embodiments 

of the present invention can be implemented; 

[0019] FIG. 3 shows a free space pointing device according to an exemplary 

embodiment of the present invention; 

[0020] . FIG. 4 illustrates a cutaway view of the free space pointing device in FIG. 4 
including two rotational sensors and one accelerometer; 

[0021] FIG. 5 is a block diagram illustrating processing of data associated with free 

space pointing devices according to an exemplary embodiment of the present invention; 
[0022] FIGS. 6(a) -6(d) illustrate the effects of tilt; 
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[0023] FIG. 7 depicts a hardware architecture of a free space pointing device 

according to an exemplary embodiment of the present invention; 

[0024] FIG. 8 is a state diagram depicting a stationary detection mechanism according 

to an exemplary embodiment of the present invention; 

[0025] FIG. 9 is a block diagram illustrating a system for removing unintentional 

movement from detected motion according to an exemplary embodiment of the present 
invention; and 

[0026] FIG. 1 0 depicts various examples of detected movement associated with fine 

button clicking and coarse button clicking. 



DETAILED DESCRIPTION 
[0027] The following detailed description of the invention refers to the accompanying 

drawings. The same reference numbers in different drawings identify the same or similar 
elements. Also, the following detailed description does not limit the invention. Instead, the 
scope of the invention is defined by the appended claims. 

[0028] In order to provide some context for this discussion, an exemplary aggregated 

media system 200 in which the present invention can be implemented will first be described 
with respect to Figure 2. Those skilled in the art will appreciate, however, that the present 
invention is not restricted to implementation in this type of media system and that more or 
fewer components can be included therein. Therein, an input/output (I/O) bus 210 connects 
the system components in the media system 200 together. The I/O bus 210 represents any of 
a number of different of mechanisms and techniques for routing signals between the media 
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system components. For example, the I/O bus 210 may include an appropriate number of 
independent audio "patch" cables that route audio signals, coaxial cables that route video 
signals, two-wire serial lines or infrared or radio frequency transceivers that route control 
signals, optical fiber or any other routing mechanisms that route other types of signals. 
[0029] In this exemplary embodiment, the media system 200 includes a 

television/monitor 212, a video cassette recorder (VCR) 214, digital video disk (DVD) 
recorder/playback device 216, audio/video tuner 218 and compact disk player 220 coupled to 
the I/O bus 210. The VCR 214, DVD 216 and compact disk player 220 may be single disk or 
single cassette devices, or alternatively may be multiple disk or multiple cassette devices. 
They may be independent units or integrated together. In addition, the media system 200 
includes a microphone/speaker system 222, video camera 224 and a wireless I/O control 
device 226. According to exemplary embodiments of the present mvention, the wireless I/O 
control device 226 is a free space pointing device according to one of the exemplary 
embodiments described below. The wireless I/O control device 226 can communicate with 
the entertainment system 200 using, e.g., an IR or RF transmitter or transceiver. 
Alternatively, the I/O control device can be connected to the entertainment system 200 via a 
wire. 

[0030] The entertainment system 200 also includes a system controller 228. 

According to one exemplary embodiment of the present invention, the system controller 228 
operates to store and display entertainment system data available from a plurality of 
entertainment system data sources and to control a wide variety of features associated with 
each of the system components. As shown in Figure 2, system controller 228 is coupled, 
either directly or indirectly, to each of the system components, as necessary, through I/O bus 
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210. In one exemplary embodiment, in addition to or in place of I/O bus 210, system 
controller 228 is configured with a wireless communication transmitter (or transceiver), 
which is capable of communicating with the system components via IR signals or RF signals. 
Regardless of the control medium, the system controller 228 is configured to control the 
media components of the media system 200 via a graphical user interface described below. 
[0031] As further illustrated in Figure 2, media system 200 may be configured to 

receive media items from various media sources and service providers. In this exemplary 
embodiment, media system 200 receives media input from and, optionally, sends information 
to, any or all of the following sources: cable broadcast 230, satellite broadcast 232 (e.g., via a 
satellite dish), very high fi-equency (VHF) or ultra high frequency (UHF) radio frequency 
communication of the broadcast television networks 234 (e.g., via an aerial antenna), 
telephone network 236 and cable modem 238 (or another source of Internet content). Those 
skilled in the art will appreciate that the media components and media sources illustrated and 
described with respect to Figure 2 are purely exemplary and that media system 200 may 
include more or fewer of both. For example, other types of inputs to the system include 
AM/FM radio and satellite radio. 

[0032] More details regarding this exemplary entertainment system and fi-ameworks 

associated therewith can be found in the above-incorporated by reference U.S. Patent 
Application "A Control Framework with a Zoomable Graphical User Interface for 
Organizing, Selecting and Launching Media Items". Alternatively, remote devices in 
accordance with the present invention can be used in conjunction with other systems, for 
example computer systems including, e.g., a display, a processor and a memory system or 
with various other systems and applications. 
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[0033] As mentioned in tiie Background section, remote devices which operate as free 

space pointers are of particular interest for the present specification. Such devices enable the 
translation of movement, e.g., gestures, into commands to a user interface. An exemplary 
free space pointing device 400 is depicted in Figure 3. Therein, user movement of the free 
space pointing can be defined, for example, in terms of a combination of x-axis attitude (roll), 
y-axis elevation (pitch) and/or z-axis heading (yaw) motion of the free space pointing device 
400. In addition, some exemplary embodiments of the present invention can also measure 
linear movement of the free space pointing device 400 along the x, y, and z axes to generate 
cursor movement or other user interface commands. In the exemplary embodiment of Figure 
3, the free space pointing device 400 includes two buttons 402 and 404 as well as a scroll 
wheel 406, although other exemplary embodiments will include other physical 
configurations. According to exemplary embodhnents of the present invention, it is 
anticipated that free space pointing devices 400 will be held by a user in front of a display 
408 and that motion of the free space pointing device 400 will be translated by the free space 
pointing device into output which is usable to interact with the information displayed on 
display 408, e.g., to move the cursor 410 on the display 408. For example, rotation of the 
free space pointing device 400 about'the y-axis can be sensed by the free space pointing 
device 400 and translated into an output usable by the system to move cursor 410 along the y2 
axis of the display 408. Likewise, rotation of the free space pointmg device 408 about the z- 
axis can be sensed by the free space pointing device 400 and translated into an output usable 
by the system to move cursor 410 along the X2 axis of the display 408. It will be appreciated 
that the output of free space pointing device 400 can be used to interact with the display 408 
in a number of ways other than (or in addition to) cursor movement, for example it can 
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control cursor fading, volume or media transport (play, pause, fast-forward and rewind). 
Input commands may include operations in addition to cursor movement, for example, a 
zoom in or zoom out on a particular region of a display. A cursor may or may not be visible. 
Similarly, rotation of the free space pointing device 400 sensed about the x-axis of free space 
pointing device 400 can be used in addition to, or as an alternative to, y-axis and/or z-axis 
rotation to provide input to a user interface. 

[0034] According to one exemplary embodiment of the present invention, two 

rotational sensors 502 and 504 and one accelerometer 506 can be employed as sensors in free 
space pointing device 400 as shown in Figure 4. The rotational sensors 502 and 504 can, for 
example, be implemented using ADXRS150 or ADXRS401 sensors made by Analog 
Devices. It will be/appreciated by those skilled in the art that other types of rotational sensors 
can be employed as rotational sensors 502 and 504 and that the ADXRS150 and ADXRS401 
are purely used as an illustrative example. Unlike traditional gyroscopes, the ADXRS150 
rotational sensors use MEMS technology to provide a resonating mass which is attached to a 
frame so that it can resonate only along one direction. The resonating mass is displaced when 
the body to which the sensor is affixed is rotated around the sensor's sensing axis. This 
displacement can be measured using the Coriolis acceleration effect to determme an angular 
velocity associated with rotation along the sensing axis. If the rotational sensors 502 arid 504 
have a single sensing axis (as for example the ADXRSlSOs), then they can be mounted in the 
free space pointing device 400 such that their sensing axes, are aligned with the rotations to be 
measured. For this exemplary embodiment of the present invention, this means that 
rotational sensor 504 is mounted such that its sensing axis is parallel to the y-axis and that 
rotational sensor 502 is mounted such that its sensing axis is parallel to the z-axis as shown in 
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Figure 4. Note, however, that aligning the sensing axes of the rotational sensors 502 and 504 
parallel to the desired measurement axes is not required since exemplary embodiments of the 
present invention also provide techniques for compensating for offset between axes. 
[0035] One challenge faced in implementing exemplary free space pointing devices 

400 in accordance with the present invention is to employ components, e.g., rotational 
sensors 502 and 504, which are not too costly, while at the same time providing a high degree 
of correlation between movement of the free space pointing device 400, a user's expectation 
regardmg how the user interface will react to that particular movement of the free space 
pointing device and actual user interface performance in response to that movement. For 
example, if the free space pointing device 400 is not moving, the user will likely expect that 
the cursor ought not to be drifting across the screen. Likewise, if the user rotates the free 
space pointing device 400 purely around the y-axis, she or he would likely not expect to see 
the resulting cursor movement on display 408 contain any significant X2 axis coniponent. To 
achieve these, and other, aspects of exemplary embodiments of the present invention, various 
measurements and calculations are performed by the handheld device 400 which are used to 
adjust the outputs of one or more of the sensors 502, 504 and 506 and/or as part of the input 
used by a processor to determme an appropriate output for the user interface based on the 
outputs of the sensors 502, 504 and 506. These measurements and calculations are used to 
compensate for factors which fall broadly into two categories: (1) factors which are intrinsic 
to the free space pointing device 400, e.g., errors associated with the particular sensors 502, 
504 and 506 used in the device 400 or the way in which the sensors are mounted in the device 
400 and (2) factors which are not mtrinsic to the free space pointing device 400, but are 
instead associated with the manner in which a user is using the free space pointing device 
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400, e.g., linear acceleration, tilt and tremor. Exemplary techniques for handling each of 
these effects are described below. 



pointing devices according to exemplary embodiments of the present invention is illustrated 
in Figure 5. The rotational sensors 502 and 504, as well as the accelerometer 506, produce 
analog signals which are sampled periodically, e.g., 200 samples/second. For the purposes of 
this discussion, a set of these uiputs shall be referred to usmg the notation (x, y, z, ay, az), 
wherem x, y, z are the sampled output values of the exemplary three-axis accelerometer 506 
which are associated with acceleration of the free space pointing device in the x-axis, y-axis 
and z-axis directions, respectively, ay is a the sampled output value from rotational sensor 
502 associated with the rotation of the free space pointing device about the y-axis and az is 
the sampled output value from rotational sensor 504 associated with rotation of the free space 
pomting device 400 about the z-axis. 

[0037] The output from the accelerometer 506 is provided and, if the accelerometer 

506 provides analog output, then the output is sampled and digitized by an AID converter 
(not shown) to generate sampled accelerometer output 602. The sampled output values are 
converted from raw units to units of acceleration, e.g., gravities (g), as indicated by 
conversion function 604. The acceleration calibration block 606 provides the values used for 
the conversion flmction 604. This calibration of the accelerometer output 602 can include, 
for example, compensation for one or more of scale, offset and axis misalignment error 
associated with the accelerometer 506. Exemplary conversions for the accelerometer data 
can be performed using the following equation: 



[0036] 



A process model 600 which describes the general operation of free space 



A = S*((M-P).*G(T)) 



(1) 
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wherein M is a 3x1 column vector composed of the sampled output values (x, y, z), P is a 3x1 
column vector of sensor offsets, and S is a 3x3 matrix that contains both scale, axis 
misalignment, and sensor rotation compensation. G(T) is a gain factor that is a function of 
temperature. The "*" operator represents matrix multiplication and the ".*" operator 
represents element multiplication. The exemplary accelerometer 506 has an exemplary full 
range of +/- 2g. Sensor offset, P, refers to the sensor output, M, for an accelerometer 
measurement of Og. Scale refers to the conversion factor between the sampled unit value and 
g. The actual scale of any given accelerometer sensor may deviate from these nominal scale 
values due to, e.g., manufacturing variances. Accordingly the scale factor m the equations 
above will be proportional to this deviation. 

[0038] Accelerometer 506 scale and offset deviations can be measured by, for 

example, applying Ig of force along one an axis and measuring the result, Rl. Then a -Ig 
force is applied resulting in measurement R2. The individual axis scale, s, and the individual 
axis offset, p, can be computed as follows: 

s = (Rl-R2)/2 (2) 
p = (Rl+R2)/2 (3) 
In this simple case, P is the column vector of the p for each axis, and S is the diagonal matrix 
of the 1/s for each axis. 

[0039] However, in addition to scale and offset, readings generated by accelerometer 
506 may also suffer from cross-axes effects. Cross-axes effects include non-aligned axes, 
e.g., wherein one or more of the sensing axes of the accelerometer 506 as it is mounted in the 
free space pointing device 400 are not aligned with the corresponding axis m the inertia! 
frame of reference, or mechanical errors associated with the machining of the accelerometer 
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506 itself, e.g., wherein even though the axes are properly aligned, a purely y-axis 
acceleration force may result in a sensor reading along the z-axis of the accelerometer 506. 
Both of these effects can also be measured and added to the calibration performed by 
function 606, / 

[0040] The accelerometer 506 serves several purposes in exemplary free space 

pointing devices according to exemplary embodiments of the present invention. For 
example, if rotational sensors 502 and 504 are implemented using the exemplary Coriolis 
effect rotational sensors described above, then the output of the rotational sensors 502 and 
504 will vary based on the linear acceleration experienced by each rotational sensor. Thus, 
one exemplary use of the accelerometer 506 is to compensate for fluctuations in the readings 
generated by the rotational sensors 502 and 504 which are caused by variances in luiear 
acceleration. This can be accomplished by multiplying the converted accelerometer readings 
by a gain matrix 610 and subtracting (or adding) the results from (or to) the corresponding 
sampled rotational sensor data 612. For example, the sampled rotational data ay from 
rotational sensor 502 can be compensated for linear acceleration at block 614 as: 

ay' = ay-C*A (4) 
wherein C is the 1x3 row vector of rotational sensor susceptibility to linear acceleration along 
each axis given in units/g and A is the calibrated linear acceleration. Similarly, linear 
acceleration compensation for the sampled rotational data az from rotational sensor 504 can 
be provided at block 614. The gain matrices, C, vary between rotational sensors due to 
manufacturing differences. C may be computed using the average value for many rotational 
sensors, or it may be custom computed for each rotational sensor. 
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[0041] Like the accelerometer data, the sampled rotational data 612 is then converted 

from a sampled unit value into a value associated with a rate of angular rotation, e.g., 
radians/s, at function 616. This conversion step can also include calibration provided by 
function 618 to compensate the sampled rotational data for, e.g., scale and offset. 
Conversion/calibration for both ay and az can be accomplished using, for example, the 
following equation: 

arad/s = (a'-offset(T))*scale + dOffset (5) 
wherein a' refers to the value being converted/calibrated, of6et(T) refers to an offeet value associated 
with temperature, scale refers to the conversion factor between the sampled unit value and 
rad/s, and dOffset refers to, a dynamic offset value. Equation (5) may be implemented as a 
matrix equation m which case all variables are vectors except for scale. In matrix equation 
form, scale corrects for axis misalignment and rotational offset factors. Each of these 
variables is discussed in more detail below. 

[0042] The offset values offset(T) and dOffset can be determined in a number of 

different ways. When the free space pointing device 400 is not being rotated m, for example, 

the y-axis direction, the sensor 502 should output its offset value. However, the offset can be 

highly affected by temperature, so this offset value will likely vary. Offset temperature 

calibration may be performed at the factory, in which case the value(s) for offset(T) can be 

preprogrammed into the handheld device 400 or, alternatively, offset temperature calibration 

may also be learned dynamically during the lifetime of the device. To accomplish dynamic 

offset compensation, an input from a temperature sensor 619 is used in rotation calibration 

function 618 to compute the current value for offset(T). The offset(T) parameter removes the 

majority of offset bias from the sensor readings. However, negating nearly all cursor drift at 

zero movement can be useful for producing a high-performance pointing device. Therefore, 
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the additional factor dOffset, can be computed dynamically while the free space pointing 
device 400 is in use. The stationary detection function 608 determines when the handheld is 
most likely stationary and when the offset should be recomputed. Exemplary techniques for 
implementing stationary detection function .608, as well as other uses therefore, are described 
below. 

[0043] An exemplary implementation of dOffset computation employs calibrated 

sensor outputs which are low-pass filtered. The stationary output detection function 608 
provides an indication to rotation calibration function 61 8 to trigger computation of, for 
example, the mean of the low-pass filter output. The stationary output detection function 608 
can also control when the newly computed mean is factored into the existing value for 
dOffset. Those skilled in the art will recognize that a multitude of different techniques can be 
used for computing the new value for dOffset from the existing value of dOffset and the new 
mean including, but not limited to, simple averaging, low-pass filtering and Kalman filtering. 
Additionally, those skilled in the art will recognize that numerous variations for offset 
compensation of the rotational sensors 502 and 504 can be employed. For example, the 
offset(T) function can have a constant value (e.g., invariant with temperature), more than two 
offset compensation values can be used and/or only a single offset value can be 
computed/used for offset compensation. 

[0044] After conversion/calibration at block 616, the inputs from the rotational 

sensors 502 and 504 can be further processed to rotate those inputs into an inertial frame of 
reference, i.e., to compensate for tih associated with the manner in which the user is holding 
the free space pointing device 400, at function 620. Tilt correction is another significant 
aspect of some exemplary embodiments of the present invention as it is intended to 
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compensate for differences in usage patterns of free space pointing devices according to the 
present invention. More specifically, tilt correction according to exemplary embodiments of 
the present invention is intended to compensate for the fact that users will hold pointing 
devices in their hands at different x-axis rotational positions, but that the sensing axes of the 
rotational sensors 502 and 504 in the free space pointing devices 400 are fixed. It is desirable 
that cursor translation across display 408 is substantially insensitive to the way in which the 
user grips the free space pointing device 400, e.g., rotating the free space pointing device 400 
back and forth in a manner generally corresponding to the horizontal dimension (x2-axis) of 
the display 508 should result in cursor translation along the X2-axis, while rotating the free 
space pointing device up and down in a manner generally corresponding to the vertical 
dimension (y2-axis) of the display 508 should result in cursor translation along the y2-axis, 
regardless of the orientation in which the user is holding the free space pointing device 400, 
[0045] To better understand the need for tilt compensation according to exemplary 

embodiments of the present invention, consider the example shown in Figure 6(a). Therein, 
the user is holding free space pointing device 400 in an exemplary inertial frame of reference, 
which can be defined as having an x-axis rotational value of 0 degrees, e.g., the inertial frame 
of reference can that in which the free space device has its bottom substantially parallel to a 
floor of a room in which, e.g., a television is located. The inertial frame of reference can, 
purely as an example, correspond to the orientation illusfrated in Figure 6(a) or it can be 
defined as any other orientation. Rotation of the free space pointing device 400 in either the 
y-axis or z-axis directions will be sensed by rotational sensors 502 and 504, respectively. 
For example, rotation of the free space pointing device 400 around the z-axis by an amount 
Az as shown in Figure 6(b) will result in a corresponding cursor translation Ax2 in the X2 axis 
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dimension across the display 408(i,e., the distance between the dotted version of cursor 410 
and the undotted version). 

[0046] If, on the other hand, the user holds the free space pointing device 400 in a 

different orientation, e.g., with some amount of x-axis rotation relative to the inertial frame of 
reference, then the information provided by the sensors 502 and 504 would not (absent tilt 
compensation) provide an accurate representation of the user's intended interface actions. 
For example, referring to Figure 6(c), consider a situation wherein the user holds the free 
space pointing device 400 with an x-axis rotation of 45 degrees relative to the exemplary 
inertial frame of reference as illustrated in Figure 6(a). Assuming the same z-axis rotation Az 
is imparted to the free space pointing device 400 by a user as in the example of Figure 6(b), 
the cursor 410 will instead be translated m both the X2-axis du-ection and the ya-axis direction 
as shown in Figure 6(d). This is due to the fact that the sensing axis of rotational sensor 502 
is now oriented between the y-axis and the z-axis (because of the orientation of the device in 
the user's hand). Similarly, the sensing axis of the rotational sensor 504 is also oriented 
between the y-axis and the z-axis (although in a different quadrant). In order to provide an 
interface which is transparent to the user m terms of how the free space pomting device 400 
is held, tilt compensation according to exemplary embodiments of the present invention 
translates the readings output from rotational sensors 502 and 504 back into the inertial frame 
of reference as part of processing the readings from these sensors mto information indicative 
of rotational motion of the free space pointing device 400. 

[0047] According to exemplary embodiments of the present invention, returning to 

Figure 5, this can be accomplished by determining the tilt of the free space pointing device 
400 using the inputs y and z received from accelerometer 506 at fimction 622. More 

-23- 



wo 2005/109215 



PCT/US2005/015068 



0320-058 



specifically, after the acceleration data is converted and calibrated as described above, it can 
be low pass filtered at LPF 624 to provide an average acceleration (gravity) value to the tilt 
determination function 622. Then, tilt 9 can be calculated in function 622 as: 



The value 9 can be numerically computed as atan2(y,z) to prevent division by zero and give 
the correct sign. Then, function 620 can perform the rotation R of the converted/calibrated 
inputs ay and az using the equation: 



to rotate the converted/calibrated inputs ay and az to compensate for the tilt Q. Tilt 
compensation as described in this exemplary embodiment is a subset of a more general 
technique for translating sensor readings from the body frame of reference into a user's frame 
of reference according to another exemplary embodiment of the present invention which is 
described below. 

[0048] Once the calibrated sensor readings have been compensated for linear 

acceleration, processed into readings indicative of angular rotation of the free space pointing 
device 400, and compensated for tilt, post-processing can be performed at blocks 626 and 
628. Exemplary post-processing can include compensation for various factors such as human 
tremor. Although tremor may be removed using several different methods, one way to 
remove tremor is by using hysteresis. The angular velocity produced by rotation function 
620 is integrated to produce an angular position. Hysteresis of a calibrated magnitude is then 
applied to the angular position. The derivative is taken of the output of the hysteresis block 
to again yield an angular velocity. The resulting output is then scaled at function 628 (e.g.. 
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based on the sampling period) and used to generate a result within the interface, e.g., 
movement of a cursor 41 0 on a display 408. 

[0049] Having provided a process description of exemplary free space pointing 

devices according to the present invention, Figure 7 illustrates an exemplary hardware 
architecture. Therein, a processor 800 communicates with other elements of the free space 
pointmg device including a scroll wheel 802, JTAG 804, LEDs 806, switch matrix 808, IR 
photodetector 810, rotational sensors 812, accelerometer 814.and transceiver 816. The scroll 
wheel 802 is an optional input component which enables a user to provide input to the 
interface by rotating the scroll wheel 802 clockwise or counterclockwise. JTAG 804 
provides the programming and debugging interface to the processor. LEDs 806 provide 
visual feedback to a user, for example, when a button is pressed. Switch matrix 808 receives 
inputs, e.g., indications that a button on the free space pointmg device 400 has been 
depressed or released, that are then passed on to processor 800. The optional IR 
photodetector 810 can be provided to enable the exemplary free space pomting device to 
learn IR codes from other remote controls. Rotational sensors 812 provide readings to 
processor 800 regarding, e.g., the y-axis and z-axis rotation of the free space pointing device 
as described above. Accelerometer 814 provides readings to processor 800 regarding the 
linear acceleration of the free space pointing device 400 which can be used as described 
above, e.g., to perform tih compensation and to compensate for errors which linear 
acceleration mtroduces into the rotational readings generated by rotational sensors 812. 
Transceiver 816 is used to communicate information to and from free space pointing device 
400, e.g., to the system confroller 228 or to a processor associated with a computer. The 
transceiver 816 can be a wireless transceiver, e.g., operating in accordance with the Bluetooth 
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Standards for short-range wireless communication or an infrared transceiver. Alternatively, 
free space pointing device 400 can communicate with systems via a wireline connection. 
[0050] In the exemplary embodiment of Figure 4, the free space pointing device 400 

includes two rotational sensors 502 and 504, as well as an accelerometer 506. However, 
according to another exemplary embodunent of the present invention, a free space pointing 
device can alternatively include just one rotational sensor, e.g., for measuring angular 
velocity in the z-axis direction, and an accelerometer. For such an exemplary embodiment, 
similar functionality to that described above can be provided by using the accelerometer to 
determine the angular velocity along the axis which is not sensed by the rotational sensor. 
For example, rotational velocity around the y-axis can be computed using data generated by 
the accelerometer and calculating: 




(9) 



In addition, the parasitic acceleration effects that are not measured by a rotational sensor 
should also be removed. These effects include actual linear acceleration, acceleration 
measured due to rotational velocity and rotational acceleration, and acceleration due to 
human tremor. 

[0051] Stationary detection function 608, mentioned briefly above, can operate to 

determine whether the free space pointing device 400 is, for example, either stationary or 

active (moving). This categorization can be performed in a number of different ways. One 

way, according to an exemplary embodiment of the present invention, is to compute the 

variance of the sampled input data of all inputs (x, y, z, ay, az) over a predetermined window, 

e.g., every quarter of a second. This variance is then compared with a threshold to classify 

the free space pointing device as either stationary or active. 
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[0052] Another stationary detection teclmique according to exemplary embodiments 
of the present invention involves transforming the inputs mto the frequency domain by, e.g., 
performmg a Fast Fourier Transform (FFT) on the input data. Then, the data can be analyzed 
using, e.g., peak detection methods, to determine if the free space pointing device 400 is 
either stationary or active. Additionally, a third category can be distinguished, specifically 
the case where a user is holding the free space pointing device 400 but is not moving it (also 
■referred to herein as the "stable" state. This third category can be distinguished from 
stationary, (not held) and active by detecting the small movement of the free space pointing 
device 400 introduced by a user's hand tremor when the free space pointing device 400 is 
being held by a user. Peak detection can also be used by stationary detection fimction 608 to 
make this determination. Peaks within the range of human tremor frequencies, e.g., 
nominally 8-12 Hz, will typically exceed the noise floor of the device (experienced when the 
device is stationary and not held) by approximately 20 dB. 

[0053] In the foregomg examples, the variances in the frequency domain were sensed 

within a particular frequency range, however the actual frequency range to be monitored and 
used to characterize the status of the free space pointing device 400 may vary. For example, 
the nominal tremor frequency range may shift based on e.g., the ergonomics and weight of 
the free space pointing device 400, e.g., from 8-12 Hz to 4-7 Hz. 
[0054] Accordmg to another exemplary embodiment of the present invention, 

stationary detection mechanism 608 can include a state machine. An exemplary state 
machine is shown in Figure 8. Therein, the ACTIVE state is, in this example, the default 
state during which the free space pointing device 400 is moving and being used to, e.g., 
provide inputs to a user interface. The free space pointing device 400 can enter the ACTIVE 
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State on power-up of the device as indicated by the' reset input. If the free space pointing 
device 400 stops moving, it may then enter the INACTIVE state. The various state 
transitions illustrated in Figure 8 can be triggered by any of a number of different criteria 
including, but not limited to, data output from one or both of the rotational sensors 502 and 
504, data output from the accelerometer 506, time domain data, frequency domain data or 
any combination thereof. State transition conditions will be generically referred to herein 
using the convention "ConditionsateA^stateB • For example, the free space pointing device 400 
will transition from the ACTIVE state to the INACTIVE state when conditionactive^macave 
occurs. For the sole purpose of illustration, consider that conditionactive^inactive can, in an 
exemplary free space pointing device 400, occur when mean and/or standard deviation 
values from both the rotational sensor(s) and the accelerometer fall below first predetermined 
threshold values for a first predetermined time period. When in the ACTIVE state, data 
received from the motion sensors (e.g., rotational sensor(s) and/or accelerometer) can be 
separated into first data associated with intentional movement infroduced by a user and 
second data associated with unintentional movement introduced by a user (tremor) using one 
or more processing techniques such as linear filtering, Kalman filtering, Kalman smoothing, 
state-space estimation, Expectation-Maxunization, or other model-based techniques. The 
first data can then be fiirther processed to generate an output associated with the intended 
movement of the handheld device (e.g., to support cursor movement) while the second data 
can be used as fremor input for, e.g., user identification, as described in more detail in the 
above-incorporated by reference patent application. 

[0055] State fransitions can be determined by a number of different conditions based 

upon the interpreted sensor outputs. Exemplary condition metrics include the variance of the 
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interpreted signals over a time window, the threshold between a reference value and the 
interpreted signal over a time wmdow, the threshold between a reference value and the 
filtered interpreted signal over a tune window, and the threshold between a reference value 
and the interpreted signal fi-om a start tune can be used to determine state transitions. All, or 
any combination, of these condition metrics can be used to trigger state transitions. 
Alternatively, other metrics can also be used. Accordmg to one exemplary embodunent of 
the present invention, a transition fi-om the INACTIVE state to the ACTIVE state occurs 
either when (1) a mean value of sensor output(s) over a time window is greater than 
predetermmed threshold(s) or (2) a variance of values of sensor output(s) over a time window 
is greater than predetermined threshold(s) or (3) an instantaneous delta between sensor values 
is greater than a predetermined threshold. 

[0056] The INACTIVE state enables the stationary detection mechanism 608 to 

distinguish between brief pauses durmg which the free space pointing device 400 is still 
being used, e.g., on the order of a tenth of a second, and an actual transition to either a stable 
or stationary condition. This protects against the fiinctions which are performed during the 
STABLE and STATIONARY states, described below, fi-om inadvertently being performed 
when the free space pointing device is being used. The free space pointing device 400 will 
transition back to the ACTIVE state when conditionjnactive^active occurs, e.g., if the free space 
pointing device 400 starts moving again such that the measured outputs from the rotational 
sensor(s) and the accelerometer exceeds the fust threshold before a second predetermined 
time period in the INACTIVE state elapses. 

[0057] The free space pointing device 400 will transition to either the STABLE state 

or the STATIONARY state after the second predetermined time period elapses. As 
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mentioned earlier, the STABLE state reflects the characterization of the free space pointing 
device 400 as being held by a person but being substantially unmoving, while the 
STATIONARY state reflects a characterization of the free space pointing device as not being 
held by a person. Thus, an exemplary state machine according to the present invention can 
provide for a transition to the STABLE state after the second predetermined time period has 
elapsed if minimal movement associated with hand tremor is present or, otherwise, transition 
to the STATIONARY state. 

[0058] The STABLE and STATIONARY states define times during which the free 

space pomting device 400 can perform various functions. For example, since the STABLE 
state is intended to reflect times when the user is holding the free space pointing device 400 
but is not moving it, the device can record the movement of the free space pointing device 
400 when it is in the STABLE state e.g., by storing outputs from the rotational sensor(s) 
and/or the accelerometer while m this state. These stored measurements can be used to 
determine a tremor pattern associated with a particular user or users as described below. 
Likewise, when in the STATIONARY state, the free space pointing device 400 can take 
readings from the rotational sensors and/or the accelerometer for use in compensating for 
offset as described above. 

[0059] If the free space pointing device 400 starts to move while in either the 

STABLE or STATIONARY state, this can trigger a return to the ACTIVE state. Otherwise, 
after measurements are taken, the device can transition to the SLEEP state. While in the 
sleep state, the device can enter a power down mode wherein power consumption of the free 
space pointing device is reduced and, e.g., the sampling rate of the rotational sensors and/or 
the accelerometer is also reduced. The SLEEP state can also be entered via an external 
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command so that the user or another device can command the free space pointing device 400 
to enter the SLEEP state. 

[0060] Upon receipt of another command, or if the free space pointing device 400 

begins to move, the device can transition from the SLEEP state to the WAKEUP state. Like 
the INACTIVE state, the WAKEUP state provides an opportunity for the device to confirm 
that a transition to the ACTIVE state is justified, e.g., that the free space pointing device 400 
was not inadvertently jostled. 

[0061] The conditions for state transitions may be symmetrical or may differ. Thus, 

the threshold associated with the conditionactive^imctive may be the same as (or different from) 
the threshold(s) associated with the conditioni„active^active- This enables free space pointing 
devices according to the present invention to more accurately capture user input. For 
example, exemplary embodiments which include a state machine implementation allow, 
among other things, for the threshold for transition into a stationary condition to be different 
than the threshold for the transition out of a stationary condition. 
[0062] Entering or leaving a state can be used to trigger other device functions as 

well. For example, the user interface can be powered up based on a transition from any state 
to the ACTIVE state. Conversely, the free space pointing device and/or the user mterface can 
be turned off (or enter a sleep mode) when the free space pointing device transitions from 
ACTIVE or STABLE to STATIONARY or INACTIVE. Alternatively, the cursor 410 can be 
displayed or removed from the screen based on the fransition from or to the, stationary state of 
the free space pointing device 400. 

[0063] According to exemplary embodiments of the present invention, techniques for 

processing sensed motion remove undesirable effects due to, for example, other user-device 
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interactions, such as button actuation, and/or tremor. Generally, as referred to in Figure 9, the 
input to the system is himian movement of the handheld, free space pointing device. This 
movement is sensed by the device (block 910) and processed into representative motion, e.g., 
at block 912, detailed examples of which are described above. It should be noted, however 
that these exemplary embodiments of the present invention are not limited to application in 
the exemplary handheld, free space pointing device 400 described above and are expressly 
intended to include other handheld devices, e.g., free space pointmg devices using other types 
of motion sensors. The representative motion is then converted into a meaningful 
representation (block 914) that is processed by exemplary "human factors" techniques 
according to exemplary embodiments of the present invention at block 916. In the exemplary 
embodiment described herem, the output of human factors processing 916 is then mapped 
into, for example, 2D pointer movement. The processed movement is then output by the 
handheld device, an example of which is data that can be used to control on-screen pointer 
movement. 

[0064] This exemplary embodiment of the present invention includes a variety of 

different techniques for processing movement during user initiated events including button 
clicks. According to a fu-st exemplary embodunent, both a distance threshold and a time 
threshold are employed to process movement information generated by the motion sensor(s), 
e.g., rotational sensor(s), accelerometer(s), magnetometer(s), gyroscope(s), camera(s), or any 
combination thereof, etc., after a user action, e.g., a button click, has occurred. Distance 
alone may not be sufficient to yield both a stable and a responsive pointer during a button 
click. When a button press is detected by the hand held device, the output pointer movement 
from 916 is suppressed until either the distance exceeds a distance threshold or the amount of 
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elapsed time exceeds a time threshold. Either or both of the distance and time thresholds may 
be different for, e.g., a button press action and button release action. The exemplary button 
processing may optionally be disabled by sending a command to the handheld. In this 
context, distance refers to physical device movement distance, an angle moved as detected by 
the sensor(s), and/or the distance the pointer moves (processed former data). 
[0065] Different buttons can also have different thresholds from one another. The 

amount of movement that the handheld experiences during a button click depends upon a 
number of factors including, but not limited to, the user, the button actuation force, the button 
travel, and the location of the button relative to the handheld center of support (normally a 
user's hand). The button movement processing parameters may be set individually to 
optimize the performance for each button. In addition, the parameters can be learned based 
upon the session history or based upon the user if the user is known to the system. 
[0066] Additionally, the human factors processing function 916 may store and track 

the past movement history of the handheld device. For example, when the handheld device 
detects that a button has been pressed, a processing unit in the handheld device can back up to 
the time before the user initiated the button event. Physically actuating the button takes a 
non-finite, measurable amount of time that can be determined by psychological testing and 
dynamic device measurement. When the button is actuated, the device can revert to the state 
before the button actuation occurred by deleting data samples taken from the motion 
sensor(s) during/after the button actuation occurred. Therefore, the errant movement that 
occurred during the button press will be ignored and "erased". For example, m response to a 
detected button press, the output from block 916 may change from a detected position PI 
(subsequent to a button press action) to a recalled position P2, which position P2 had 
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previously been output by block 916 a predetermined time period prior to the button press 
action detection. If the device is already processing one button action and is still suppressing 
movement when another button action occurs, it may be unnecessary for the human factors 
processing function 916 to reiterate the backing up process. 

[0067] In 'user interfaces, at least two typical types of button activations can occur. In 

the first type (fine mode clicking) shown in Figure 10, the user intends precise actuation over 
a small target and carefully aligns the device, stops movement, and then presses the button. 
In the second type (coarse mode clicking), the target is large and the user anticipates the next 
action, i.e., the user may only slow down the pointer without stopping or hovering over the 
target and may instead mtend to click on the target "on-the-fly". For fine mode clicking, the 
above-described processing technique operates to accurately remove unintentional motion 
data from the combined data output stream fi-om the motion sensor(s) in the handheld device. 
However, for the second type of movement, further enhancements may be useful to improve 
performance. 

[0068] To address the additional challenges posed by coarse mode clicking, human 

factors processing unit 916 may employ a second alternative or complementary technique. 
According to this second exemplary embodiment, movement data received fi'om the motion 
sensor(s) is processed into a movement vector and it is assumed that the user may mtend 
some movement change of the cursor or other associated output from the device durmg the 
button actuation. As known from Newton's first law, "an object in motion tends to stay in 
motion". Thus, when a button is pressed on the handheld device, it creates a high-frequency 
movement that deviates from the path. Using the motion vector and filtered movement 
information, the output from the pointing device can continue during and after the user- 
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initiated event in a manner which is consistent with previous movement history. This can be 
accomplished by adding a filter to the processing chain. The filter is designed to permit 
intended motion during and after the user-initiated event while excluding the high frequency 
movement associated with the event itself. Many processing methods such as low-pass 
filtering enable the removal of high-fi-equency components but at the expense of increased 
latency. Since latency (the time between the movement of the device and the time the pomter 
moves) may be important to the user, exemplary embodiments of the present invention can 
use an adaptive filter which is switched into the signal processing path when a user-initiated 
event is detected (e.g., based on the same signal which is used by the handheld device to 
convey the user-event to a user interface) . The adaptive filter is configured as a low-pass 
filter that attenuates sharp, high-fi-equency button presses. An additional input to the adaptive 
filtering block is an optional pre-button activity warning that occurs before the button has 
been completely debounced. The pre-button activity warning reduces the filter latency 
requirements by allowing the processing to be notified of a button event sooner than it would 
otherwise. The adaptive filter with the pre-button activity warning mmimizes the engineering 
tradeoff between intended movement and latency. 

[0069] According to yet another exemplary embodiment of the present invention, 

since the designers of fi-ee space pointing devices usually know the direction of the 
undesirable movement at design time, the direction of movement perturbation is therefore 
known based upon the ergonomics and intended use of the device. For example, the designer 
knows the direction of button actuation relative to the device. The primary movement 
consists of either linear movement parallel to the button travel vector or rotational movement 
due to the torque about the users grip. This knowledge permits the implementation of a 
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directional preference filter that includes the knowledge of the button actuation movement. 
For example, the filter design can include state-space filters such as a Kalman filter and 
adaptive filters. Such a filter detects the undesired deviation fi'om the intended path in a 
known direction during button actuation and then interpolates the desired movement path 
during button actuation. This preferential filtering yields a more responsive pointer during 
intended changes in direction while still removing the unintended button movement. One 
skilled ui the art will recognize that the state-space filter can be extended to learn the design 
parameters during the course of normal use. 

[0070] The human factors processing fiinction 916 according to exemplary 

embodiments of the present invention may implement one or both of the above-described 
techniques and, if both are used, provide a classifier to switch between the techniques. For 
example, when the first, precise type of button click is detected, the first technique can be 
used. When the second, less precise type of button click is detected, the second technique 
can be used. One classifier for switching between techniques can use the velocity of the 
handheld device at the time of button actuation or at the time just before button actuation. 
For example, if (a) the velocity of the handheld device is below a predetermined velocity 
threshold then the first technique is employed which discards motion data generated 
subsequent to a detected event until either motion sensed by the motion sensor(s) indicates 
that the handheld device has moved more than a predetermined distance threshold or a 
predetermmed time has expired, otherwise, (b) if the velocity of the handheld device is above 
the predetermined velocity, then the second technique is employed which instead filters the 
motion data generated subsequent to a detected event. 
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[0071] Button clicks or presses as referred to in the foregoing exemplary 

embodiments include, but are not limited to, both button presses and button releases. All of 
the above techniques can be applied to any known device interaction that yields undesirable 
movement, and are not limited to button clicks. For example, the above techniques can be 
applied to scroll wheel actuation, touch pad usage, or capacitive strip usage. Thus, exemplary 
embodiments of the present invention describe methods and devices for canceling unwanted 
movement that occurs based upon activating or deactivating another event. 
[0072] The parameters for the methods describe above can be adapted'to support the 

expected movement characteristic for the event, hi the exemplary embodunent, the 
parameters for button presses can be different from the parameters for button releases. In 
addition to movement cancellation, the user interface may impose or suggest additional 
constraints on user-event handling accordmg to the present invention. For example, in 
Microsoft Windows™ operating systems, if the cursor moves while the button is pressed, 
then a "drag" action is initiated. Therefore, parameters associated with motion data 
processing in response to a button press action by a user can have values which tend to 
restrict pointer movement during button actuation in order to prevent unwanted drag events. 
In contrast, cursor movement after a button release in Microsoft Windows™ operating 
systems has little effect on objects m the user interface and, therefore, exemplary motion data 
processing in accordance with the present invention can employ parameters (e.g., time and or 
distance thresholds, filter coefficients, etc.) which tend to be less restrictive of pointer 
movements as compared to the corresponding parameters associated with motion data 
processing subsequent to button presses. 



-37- 



wo 2005/109215 PCTAJS2005/015068 
0320-058 

[0073] The movement may be processed in the velocity or position domain in a 

number of different fashions to help remove unwanted button movement. Simple filtering m 
the velocity domain may be used. The filter may be an FIR or IIR filter, although these may 
introduce an undesirable amount of processing delay. An adaptiye filter can be used 
successfully without introducing too much delay. 

[0074] Exemplary embodiments of the present invention can also be implemented as 

a Kahnan filter (or extended Kabnan filter). The Kalman filter could select the most likely 
usage scenario (stationary or moving) and apply the appropriate movement. Neural networks 
can be used for the same result. Thus it will be appreciated that the present invention further 
provides a method for (a) detecting that an event has occurred, (b) inferring the user's 
intended motion and (c) interpreting the user's intended motion rather than the actual motion 
of the handheld device. The motion can be in either the 6D0F free space domain or the 
mapped 2D0F pointing domain. The 6D0F fi-ee-space domain could be in either the 
device's body frame of reference or the user's frame of reference. 
[0075] According to another exemplary embodiment of the present invention, the 

movement associated with the user-initiated event can be modeled and included explicitly in 
a motion equation which provides an output from the handheld device based on the motion 
data gathered from the motion sensor(s). More specifically, using the button press example 
of a user-initiated event, the movement caused by button presses can be modeled a priori to 
determine one or more exemplary movement amplitudes and directions associated with the 
button press action and these values can then be stored in a memory unit of the handheld 
device. Then, when a button press is detected, the processing unit of the handheld device can 
use the modeled movement in the motion equation to adapt the output such that it reflects the 
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user's intended motion rather than the movement detected by the motion sensor(s) that was 
associated with the button press. The processing unit can use the modeled values in a variety 
of different ways, e.g., subtract them from the detected motion values, attenuate the detected 
motion values based on the modeled values associated with the particular user-mitiated event, 
adjust a semantic map associated with the handheld's output, etc. 

[0076] Any or all of the parameters described above for the exemplary embodiment 

can be modified under control of the user interface. The user interface or an application may 
know a priori that the user intends fme movement and can instruct the exemplary 
embodiment to process all events as if fme movement mode was enabled. In addition the 
amount of subtraction, attenuation, or filtering may be adjusted under control of the user 
interface or application. 

[0077] Systems and methods for processing data according to exemplary 

embodiments of the present invention can be performed by one or more processors executing 
sequences of instructions contained m a memory device. Such instructions may be read into 
the memory device from other computer-readable mediums such as secondary data storage 
device(s). Execution of the sequences of instructions contained in the memory device causes 
the processor to operate, for example, as described above. In alternative embodiments, hard- 
wire circuitry may be used in place of or in combination with software instructions to 
implement the present invention. 

[0078] ■ The above-described exemplary embodiments are intended to be illustrative in 
all respects, rather than restrictive, of the present invention. Thus the present invention is 
capable of many variations in detailed implementation that can be derived from the 
description contained herein by a person skilled in the art. For example, although the 
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foregoing exemplary embodiments describe, among other things, the use of inertial sensors to 
detect movement of a device, other types of sensors (e.g., Ultrasound, magnetic or optical) 
can be used instead of, or in addition to, inertial sensors in conjunction with the afore- 
described signal processing. All such variations and modifications are considered to be 
within the scope and spirit of the present invention as defined by the following claims. No 
element, act, or instruction used in the description of the present application should be 
construed as critical or essential to the invention unless explicitly described as such. Also, as 
used herein, the article "a" is intended to include one or more items. 
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WHAT IS CLAIMED IS: 

1 . A handheld, pointing device comprising: 

a first rotational sensor for determining rotation of said pobting device about a first 
axis and generating a first rotational output associated therewith; 

a second rotational sensor for determining rotation of said pointing device about a 
second axis and generating a second rotational output associated therewith; 

an accelerometer for determining an acceleration of said pointing device and 
outputting an acceleration output associated therewith; and 

a processing unit for receiving said first and second rotational outputs and said 
acceleration output and for: 

(a) detecting a user-initiated event; 

(b) determining a velocity of said handheld pointing device based upon at 
least one of said first and second rotational outputs and said acceleration output; 

(c) if said velocity of said handheld device is below a predetermined velocity 
threshold, then discarding or attenuating data generated by said first rotational sensor, said 
second rotational sensor and said accelerometer until either said handheld device has moved 
more than a predetermined distance or a predetermined time has expired; and 

(d) otherwise, if said velocity of said handheld pomtmg device is above said 
predetermined velocity threshold, then filtering said data generated by said first rotational 
sensor, said second rotational sensor and said accelerometer to remove unintended motion 
associated with said user-initiated event. 
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2. The handheld, pointing device of claim 1, wherein said user-initiated event is one of: 
a button press, a button release, scroll wheel actuation, touch pad usage, and capacitive strip 
usage. 

3. A method for operating a free space pointing device comprising the steps of: 

sensing movement of the free space pointing device and generating an output 
associated therewith; 

detecting occurrence of an event; and 

filtering said output based on said detected event. 

4. The method of claim 3, wherein said free space pointing device includes at least one 
button and said event is a button press or button release action. 

5. The method of claim 3, wherem said step of filtering further comprises the step of: 

filtermg said output using a first filter if said event is of a first type and using a 
second filter if said event is of a second type. 

6. The method of claim 5, wherein said first type of said event is a button press and said 
second type of said event is a button release. 

7. The method of claim 3, wherein said step of sensing movement of the free space 
pointing device and generating an output associated therewith fiirther comprises: 
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, using at least one of an accelerometer, a rotational sensor, a gyroscope, a 
magnetometer, an optical sensor and a camera to sense said movement. 

8. The method of claim 3, wherein said step of filtering said output based on said 
detected event further comprises the step of: 

permitting some data generated by at least one sensor to be processed 
immediately subsequent to said detected event. 

9. A method for operating a free space pointing device comprising the steps of: 

sensing movement of the free space pointing device and generating an output 
associated therewith; 

detecting occurrence of an event; and 

discarding or attenuating said output until a predetermined time has elapsed 
after said detection of said event. 

10. The method of claim 9, wherein said free space pointing device mcludes at least one 
button and said event is a button press or button release action. 

1 1 . The method of claim 9, wherein said step of sensing movement of the free space 
pointing device and generating an output associated therewith fiirther comprises: 

using at least one of an accelerometer, a rotational sensor, a gyroscope, a 
magnetometer, an optical sensor and a camera to sense said movement. 
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12. A method for operating a free space pointing device comprising the steps of: 

sensing movement of the free space pointing device and generating an output 
associated therewith; 

detecting occurrence of an event; and 

discardmg or attenuating said output until either a predetermined time has 
elapsed after said detection of said event or said free space pointing device has moved a 
predetermined distance after said detection of said event, whichever comes first. 

13. The method of claim 12, wherein said free space pointing device includes at least one 
button and said event is a button press or button release action. 

14. The method of claim 12, wherein said step of sensing movement of the free space 
pointing device and generating an output associated therewith fiirther comprises: 

using at least one of an accelerometer, a rotational sensor, a gyroscope, a 
magnetometer, an optical sensor and a camera to sense said movement. 

15. A method for operating a free space pointing device comprising the steps of: 

(a) sensing movement of the free space pointmg device and generating an 
output associated therewith; 

(b) detecting occurrence of an event; 

(c) determining a velocity of said free space pointing device; 
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(d) if said velocity of said free space pointing device is below a predetermined 
velocity threshold, then discarding or attenuating movement information sensed after said 
detected occurrence of said event; and 

(e) otherwise, if said velocity of said free space pointing device is above said 
predetermined velocity threshold, then filtering movement information sensed after said 
detected occurrence of said event to remove unintended motion associated with said user- 
initiated event. 

16. The method of claim 15, wherein movement information is discarded until either said 
handheld device has moved more than a predetermined distance or a predetermined time has 
expired. 

17. The method of claim 1 5, wherein said free space pointing device includes at least one 
button and said event is a button press or button release action. 

1 8. The method of claim 15, wherein said step of sensing movement of the free space 
pointing device and generating an output associated therewith fijrther comprises: 

using at least one of an accelerometer, a rotational sensor, a gyroscope, a 
magnetometer, an optical sensor and a camera to sense said movement. 

19. A handheld pointing device comprising: 

at least one motion sensor for sensing movement of the handheld pointing 
device and generating an output associated therewith; 
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a processing unit for detecting occurrence of an event and for 
filtering said output based on said detected event. 

20. Tlie handiield pointing device of claim 19, wherein said handheld pointing device 
includes at least one button and said event is a button press or button release action. 

21. The handheld pointing device of claim 19, wherein said processing unit filters said 
output using a first filter if said event is of a fu-st type and using a second filter if said event is 
of a second type. 

22. The handheld pointing device of claim 21, wherein said fu-st type of said event is a 
button press and said second type of said event is a button release. 

23. The handheld pointing device of 19, wherein said at least one motion sensor includes 
at least one of an accelerometer, a rotational sensor, a gyroscope, a magnetometer, an optical 
sensor and a camera. 

24. The handheld pointmg device of claun 19, wherein said processing unit filters said 
output to permit some data generated by at least one sensor to be processed immediately 
subsequent to said detected event. 

25. A handheld pointing device comprising: 
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at least one motion sensor for sensing movement of the handheld pomting 
device and generating an output associated therewith; and 

a processing unit for detecting occurrence of an event and for discarding or 
attenuating said output until a predetermined time has elapsed after said detection of said 
event. 

26. The handheld pointing device of claim 25, wherein said handheld pointmg device 
includes at least one button and said event is a button press or button release action. 

27. The handheld pointing device of claim 26, wherein said at least one motion sensor 
includes at least one of an accelerometer, a rotational sensor, a gyroscope, a magnetometer, 
an optical sensor and a camera. 

28. A handheld pointing device comprising: 

at least one motion sensor for sensing movement of the handheld pointing 
device and generating an output associated therewith; and 

a processing unit for detecting occurrence of an event and discarding or 
attenuating said output until either a predetermined time has elapsed after said detection of 
said event or said handheld pointing device has moved a predetermmed distance after said 
detection of said event, whichever comes first. 

29. The handheld pointing device of claim 28, wherein said handheld pointing device 
includes at least one button and said event is a button press or button release action. 
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30. The handheld pointing device of claim 28, wherein said at least one motion sensor 
includes at least one of an accelerometer, a rotational sensor, a gyroscope, a magnetometer, 
an optical sensor and a camera. 

31. A handheld pointing device comprising: 

at least one motion sensor for sensing movement of the handheld pointing 
device and generating an output associated therewith; and 

a processing unit for detecting occurrence of an event, determining a velocity 
of said free space pointing device based on said sensed movement, wherein 

(a) if said velocity of said free space pointing device is below a 
predetermined velocity threshold, then said processing unit discards movement information 
sensed after said detected occurrence of said event; and 

(b) otherwise, if said velocity of said free space pointing device is 
above said predetermined velocity threshold, then said processing unit filters movement 
information sensed after said detected occurrence of said event to remove unintended motion 
associated with said user-initiated event. 

32. The handheld pointing device of claim 31, wherein said movement information is 
discarded until either said handheld device has moved more than a predetermined distance or 
a predetermined time has expired. 

33 . The handheld pointing device of claim 3 1 , wherein said handheld pointing device 
includes at least one button and said event is a button press or button release action. 
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34. The handheld pointing device of claim 3 1, wherein said at least one motion sensor 
mcludes at least one of an accelerometer, a rotational sensor, a gyroscope, a magnetometer, 
an optical sensor and a camera. 

35. A handheld pointing device comprising: 

at least one motion sensor for sensing movement of the handheld pointmg 

device and generating an output associated.therewith; 

a memory unit for storing at least one value associated with an event; and , 
a processing unit for detecting occurrence of said event and for adjusting said 

output based on said at least one stored value. 

36. The handheld pointing device of claim 35, wherein said handheld pointing device 
includes at least one button and said event is a button press or button release action. 

37. The handheld pointing device of claun 35, wherein said at least one motion sensor 
mcludes at least one of an accelerometer, a rotational sensor, a gyroscope, a magnetometer, 
an optical sensor and a camera. 
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FIG. 1 

(Prior Art) 
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FIG. 6 A 
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FIG. 6C 
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